Hierarchy in Web Page Similarity Link Analysis

Rather than using traditional text analysis to discover Web pages similar to a given page, we investigate applying link analysis. Since web pages exist in a link-rich environment, that has the potential to relate pages by any property imaginable — since links are not restricted to intrinsic properties of the page text or metadata. In particular, while Web page similarity link analysis has been explored, prior work has deliberately ignored the explicitly hierarchical host & pathname structure within URLs. To exploit this property, we generalize Kleinberg’s well-known “hubs and authorities” HITS algorithm; adapt this algorithm to accommodate hierarchical link structure; test some sample web queries; and argue that the results are potentially superior and that the algorithm itself is better motivated.